Image processing apparatus

ABSTRACT

An image processing apparatus that processes an image includes an image processor configured to: acquire a photographic image from a camera photographing a region near a host vehicle; derive a trace line showing a moving trace of a path on which the host vehicle has moved previously; derive a prediction line showing a predicted moving path of the host vehicle; generate a support image by superimposing the trace line and the prediction line on the photographic image when a wheel of the host vehicle is turned; and output the support image to a display apparatus that displays the support image.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention relates to the technology for image processing.

2. Description of the Background Art

Conventionally, an image display system which displays a surroundingimage of a vehicle, etc. is proposed. The image display system displaysa guiding line to support an operation for parking a vehicle within aparking frame by reversing the vehicle.

In the conventional technology, the guiding line is displayed and let adriver confirm a width and a traveling direction of a host vehicle. In acase where the driver turns a wheel to an opposite traveling directionto adjust a parking position, it is difficult to confirm a difference ofthe traveling direction of the host vehicle before and after turning thewheel. The driver needs to adjust the parking position again if thetraveling direction of the host vehicle after turning the wheel is sameas the traveling direction of the host vehicle before turning the wheel.However, there is a possibility that the driver moves the host vehicleto the same position after turning the wheel unless the difference ofthe traveling direction before and after turning the wheel is clarified.

SUMMARY OF THE INVENTION

According to one aspect of the invention, an image processing apparatusthat processes an image includes an image processor configured to:acquire a photographic image from a camera photographing a region near ahost vehicle; derive a trace line showing a moving trace of a path onwhich the host vehicle has moved previously; derive a prediction lineshowing a predicted moving path of the host vehicle; generate a supportimage by superimposing the trace line and the prediction line on thephotographic image when a wheel of the host vehicle is turned; andoutput the support image to a display apparatus that displays thesupport image.

Thus, the support image is generated by superimposing the trace line andthe prediction line on the photographic image when the wheel of the hostvehicle is turned to the opposite traveling direction and the generatedsupport image is displayed on the display apparatus. Therefore, a usercan grasps a difference of the traveling direction before and afterturning the wheel.

Moreover, according to another aspect of the invention, the imageprocessor is configured to acquire information as to a shift position ofthe host vehicle, and the image processor determines whether the wheelof the host vehicle has turned when the shift position changes between aposition showing a forward movement of the host vehicle and a positionshowing a backward movement of the host vehicle.

Thus, the host vehicle repeats the forward movement and the backwardmovement alternately when the wheel of the host vehicle is turned to theopposite traveling direction. Therefore, it can be determined whetherturning the wheel is carried out or not based on a change of the shiftposition.

Hence, an object of the invention is to provide a technology forspecifying the difference of the traveling direction before and afterturning a wheel by displaying the traveling direction of the hostvehicle before and after turning the wheel.

These and other objects, features, aspects and advantages of theinvention will become more apparent from the following detaileddescription of the invention when taken in conjunction with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an outline of an image display system;

FIG. 2 illustrates a photographing direction of four cameras;

FIG. 3 explains a situation of generating a support image;

FIG. 4 illustrates, an example of a support image;

FIG. 5 illustrates an example of a support image;

FIG. 6 illustrates an example of a support image;

FIG. 7 illustrates a flowchart showing a process of the image displaysystem;

FIG. 8 illustrates a flowchart showing a process of the image displaysystem;

FIG. 9 illustrates a flowchart showing a process of the image displaysystem;

FIG. 10 explains a situation of generating a support image;

FIG. 11 illustrates a flowchart showing a process of the image displaysystem;

FIG. 12 illustrates an example of support image;

FIG. 13 illustrates an example of support image;

FIG. 14 illustrates an example of support image;

FIG. 15 illustrates an example of support image;

FIG. 16 illustrates an example of support image; and

FIG. 17 illustrates an example of support image.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, some embodiments of the invention are described withreference to the drawings.

1. First Embodiment

<1-1. System Configuration>

FIG. 1 illustrates a schematic block diagram showing a configuration ofan image display system 10 in a first embodiment. The image displaysystem 10 is used for a vehicle (a car, in this embodiment). The imagedisplay system 10 generates an image showing a region near the vehicleand includes a function of displaying the image in the vehicle cabin. Auser of the image display system 10 (typically a driver) can grasp asurrounding situation of the vehicle almost in real time by using theimage display system 10. Hereinafter, the vehicle on which the imagedisplay system 10 is mounted is referred to as “host vehicle”.

As FIG. 1 shows, the image display system 10 includes plural cameras 5,an image processing apparatus 2, a display apparatus 3, and an operationbutton 4. The plural cameras 5 photograph a region near a host vehicleand acquires photographic images. The plural cameras 5 input theacquired photographic images to the image processing apparatus 2. Theimage processing apparatus 2 generates a support image to be displayedon the display apparatus 3 by using the photographic images showing theregion near the host vehicle. The display apparatus 3 displays thesupport image generated by the image processing apparatus 2. Theoperation button 4 receives a user's operation.

The plural cameras 5 include lenses and imaging devices. The pluralcameras 5 electronically acquire the photographic images showing theregion near the host vehicle. The plural cameras 5 include a frontcamera 5F, a rear camera 5B, a left side camera 5L and a right sidecamera 5R. These four cameras, 5F, 5B, 5L, 5R, are arranged in eachdifferent position in a host vehicle 9 and photograph a region near thehost vehicle in a different direction.

FIG. 2 illustrates each photographing direction of the four cameras 5F,5B, 5L, and 5R. The front camera 5F is arranged on a front end of thehost vehicle 9, having an optical axis 5Fa extending in a straightdirection of the host vehicle 9. The rear camera 5B is arranged on arear end of the host vehicle 9, having an optical axis 5Ba extending ina direction opposite to the straight direction of the host vehicle 9.The left side camera 5L is arranged on a left side mirror 93L, having anoptical axis 5La extending in a leftward direction of the host vehicle 9(an orthogonal direction to a straight direction). The right side camera5R is arranged on a right side mirror 93R, having an optical axis 5Raextending in a rightward direction of the host vehicle 9 (an orthogonaldirection to the straight direction).

The cameras 5F, 5B, 5L, and 5R include a wide lens such as a fisheyelens. The cameras 5F, 5B, 5L, and 5R include more than 180 degrees angleof view θ. The cameras 5F, 5B, 5L, and 5R can photograph all around thehost vehicle 9 by using these four cameras together.

FIG. 1 is referred. The display apparatus 3 includes a flat displaypanel such as a liquid crystal. The display apparatus 3 displays variouskinds of information and images. The display apparatus 3 is arranged ina position where the user can see the display such as an instrumentpanel of the host vehicle 9. The display apparatus 3 is arranged in asame housing with the image processing apparatus 2 and may be unifiedwith the image processing apparatus 2. However, the display apparatus 3may remain as a separate apparatus from the image processing apparatus2.

The operation button 4 is an operation member that receives the user'soperation. The operation button 4, for example, is arranged on asteering wheel of the host vehicle 9 and mainly receives the user'soperation. A touch panel of the display apparatus 3 can be used as theoperation button 4. The user performs various operations to the imagedisplay system 10 through this operation button 4 or the touch panel ofthe display apparatus 3. In a case where the user's operation is carriedout to the operation button 4, an operation signal showing an operationcontent is input to the image processing apparatus 2.

The image processing apparatus 2 is an electronic apparatus capable ofprocessing various kinds of images. The image processing apparatus 2includes an image acquisition part 21, an image composite part 22, aninformation superimposing part 23, and an image output part 24.

The image acquisition part 21 acquires photographic images acquired bythe four cameras 5F, 5B, 5L, and 5R. The image, acquisition part 21includes an image processing function which converts an analogphotographic image into a digital photographic image The imageacquisition part 21 performs a predetermined image process to theacquired photographic images and inputs the photographic images to theimage composite part 22 after performing the process.

The image composite part 22 is a hardware circuit that performs an imageprocessing to generate a composite image. The image composite part 22composes plural photographic images acquired by the plural cameras 5 andgenerates the composite image showing the region near the host vehicle 9viewing from a virtual viewpoint. The image composite part 22 projectsplural photographic images onto a virtual projection plane which isequivalent to the region near the host vehicle 9 and generates thecomposite image by use of data in the projection plane. A well-knowntechnique may be used for generating the composite image.

The information superimposing part 23 superimposes various kinds ofinformation necessary for the user on the photographic images acquiredby the image acquisition part 21 from each of the camera 5 or thecomposite image generated by the image composite part 22. Concretely,the information superimposing part 23 generates a support image bysuperimposing support information for parking (hereinafter referred toas “support information”) on the photographic images or the compositeimage when parking the host vehicle 9. The support information, forexample, is a center line, a prediction line, and a trace line. Thecenter line is a line showing an approximate center in a parking region.The prediction line is a line predicting a path on which the hostvehicle 9 will move in the future (a moving prediction line). The traceline is a line showing a trace on which the host vehicle 9 has moved inthe past (a moving trace line). Details of a technique for generatingthe support image will be described later.

The image output part 24 outputs the support image generated by theinformation superimposing part 23 to the display apparatus 3 anddisplays the support image on the display apparatus 3. Thus, an imageshowing a front portion or a rear portion of the host vehicle 9 or animage showing the region near the host vehicle 9 viewing from thevirtual viewpoint are displayed on the display apparatus 3.

The image processing apparatus 2 further includes a controller 20, amemory part 25, and a signal receiver 26. The controller 20 is amicrocomputer including a CPU, a RAM, and a ROM. The controller 20integrally controls the entire image processing apparatus 2.

The memory part 25 is a non-volatile memory such as a flash memory. Thememory part 25 stores various kinds of information. The memory part 25stores a program as firmware, or various kinds of data used by theinformation superimposing part 23 for the generation of support images,or various kinds of data used by the image composite part 22 for thegeneration of composite images, etc. Data used for the generation ofsupport images, for example, is the data showing the moving trace of thehost vehicle 9.

The signal receiver 26 receives a signal from different apparatus fromthe image processing apparatus 2 arranged in the host vehicle 9 andinputs the signal to the controller 20. The different apparatus, forexample, are a GPS (Global Positioning System) receiver 91, a shiftsensor 92, a steering sensor 94, an object detection apparatus 95 andthe like.

The signal receiver 26 receives a signal showing positional informationof the host vehicle 9 which is derived based on a signal received from aGPS satellite from the GPS receiver 91. The signal receiver 26 receivesa signal showing a shift position (a gear position) which indicates ashift lever position of a transmission apparatus of the host vehicle 9from the shift sensor 92. The signal receiver 26 receives a signalshowing a rotation direction and rotation angle of the steering of thehost vehicle 9 from the steering sensor 94. The signal receiver 26receives a signal concerning a detection result of objects existing nearthe host vehicle 9 from the object detection apparatus 95.

Various functions of the controller 20 are implemented by a performanceof arithmetic processing of the CPU based on programs stored in thememory part 25. A center line derivation part 20 a, a prediction linederivation part 20 b, a trace line derivation part 20 c, and a displaycontroller 20 d shown in FIG. 1 are implemented by the performance ofarithmetic processing of the CPU based on the programs.

The center line derivation part 20 a recognizes a region where the hostvehicle 9 can be parked and derives the approximate center part of theavailable parking region. The center line derivation part 20 a derives acenter line showing the approximate center part of the available parkingregion. The prediction line derivation part 20 b predicts the movingpath of the host vehicle 9 and derives a prediction line showing thepredicted moving path. The trace line derivation part 20 c derives themoving trace of the host vehicle 9 and derives a trace line showing thetrace on which the host vehicle 9 has moved. The display controller 20 dmakes the image output part 24 output the support image which should bedisplayed to the display apparatus 3. Details on a process of functionparts 20 a, 20 b, 20 c, and 20 d which is implemented by the controller20 will be described later.

<1-2. Generation and Display of Support Image>

With use of FIGS. 3, 4, 5, and 6, an explanation about a process ofgenerating a support image and a process of displaying the support imageis given. FIG. 3 illustrates a situation in which a host vehicle 9 isbeing parked in an available parking region P. In this embodiment, anexample of parking the host vehicle 9 by moving backward (so-calledbacking up) is explained. Concretely, in the example, after moving thehost vehicle 9 from a position 9 a to a position 9 b, a wheel of thehost vehicle 9 is turned to move to a position 9 c, and again the wheelof the host vehicle 9 is turned to park. In case of turning the wheel atthe position 9 b, from the position 9 a to the position 9 b is amovement before turning the wheel. From the position 9 b to the position9 c is a movement after turning the wheel. In case of turning the wheelat the position 9 c, from the position 9 b to the position 9 c is amovement before turning the wheel. A movement from the position 9 c is amovement after turning the wheel.

FIGS. 4, 5, and 6 illustrate examples of support images generated by useof photographic images acquired by a rear camera 5B or a front camera5F. FIG. 4 is a support image which is displayed while the host vehicle9 moves from the position 9 a. This support image is the imagesuperimposing a center line Lc and a prediction line Le on thephotographic image acquired by the rear camera 5B. FIG. 5 is a supportimage which is displayed while the host vehicle 9 moves from theposition 9 b. This support image is the image superimposing a predictionline Le and a trace line Lt on the photographic image acquired by thefront camera 5F. FIG. 6 is a support image which is displayed while thehost vehicle 9 moves from the position 9 c. This support image is theimage superimposing a center line Lc, a prediction line Le, and a traceline Lt on the photographic image acquired by the rear camera 5B.

The center line Lc is the line showing an approximate center part of theavailable parking region P. The center line Lc extends in a front-backdirection (hereinafter referred to as “parking direction”) when the hostvehicle 9 is parked in the available parking region P. The predictionline Le is the line showing a predicted moving path of the host vehicle9. The trace line Lt is the line showing a trace on which the hostvehicle 9 has moved before turning the wheel if the wheel of the hostvehicle 9 has turned. In other words, it is possible to park the hostvehicle 9 in the approximate center of the available parking region P bymoving the host vehicle 9 in a way that the prediction line Le overlapsthe center line Lc. Moreover, it is avoidable to move the host vehicle 9back to the position before turning the wheel by moving the host vehicle9 in a way that the prediction line Le does not overlap the trace lineLt.

A process of generating a support image is explained. When the hostvehicle 9 tries to start parking from the position 9 a, the center linederivation part 20 a recognizes the available parking region P based onthe photographic image acquired by the image acquisition part 21 fromthe rear camera 5B. The available parking region P is recognized basedon a white line in the photographic image or an object and availablespace in the photographic image. In addition, the available parkingregion P may be recognized based on an object and available space usinginformation acquired by an object detection apparatus 95.

The center line derivation part 20 a derives a center line of therecognized available parking region P. In other words, the center linederivation part 20 a generates a line extending in a parking directionover the approximate center part of the available parking region P.

The prediction line derivation part 20 b derives a prediction lineshowing a predicted moving path of the host vehicle 9. Concretely, theprediction line derivation part 20 b generates a line extending alongthe predicted moving path from a center part of the host vehicle 9. Aprediction of the moving path, for example, is made based on informationof a rotation direction and rotation angle of a steering acquired by asteering sensor 94.

The information superimposing part 23 generates a support image like thesupport image illustrated in FIG. 4 by superimposing and displaying thecenter line Lc and the prediction line Le on a photographic imagephotographed by the rear camera 5B. The display controller 20 d makesthe image output part 24 output the generated support image to thedisplay apparatus 3. The display apparatus 3 displays this supportimage.

When the host vehicle 9 moves from the position 9 a to the position 9 b,the host vehicle 9 stores information of the path on which the hostvehicle 9 has moved from the position 9 a (that is, a moving trace).Information about this moving path may be derived, for example, based oninformation acquired from a GPS receiver 91. Moreover, the informationabout this moving path may be derived based on plural image informationacquired at different timings.

When the host vehicle 9 tries to move to the position 9 c by turning thewheel at the position 9 b, the trace line derivation part 20 c derives atrace line based on the information of the moving path from the position9 a to the position 9 b. In other words, the trace line derivation part20 c generates a trace line before turning the wheel by connecting thepath on which the host vehicle 9 has moved before turning the wheel.

The prediction line derivation part 20 b derives a prediction linepredicting a path of the host vehicle 9 in moving from the position 9 b.In other words, the prediction line derivation part 20 b generates aline extending along the predicted path from the center part of the hostvehicle 9. The prediction of this moving path is made, similarly asdescribed above, based on the information of the rotation direction androtation angle of the steering acquired by the steering sensor 94.

The information superimposing part 23 generates a support image like thesupport image illustrated in FIG. 5 by superimposing and displaying aprediction line Le and a trace line Lt on a photographic imagephotographed by the front camera 5F. The display controller 20 d makesthe image output part 24 output the generated support image to thedisplay apparatus 3. The display apparatus 3 displays this supportimage.

When the host vehicle 9 moves from the position 9 b to the position 9 c,the host vehicle 9 stores information of the moving path from theposition 9 b (that is, a moving trace). The information about thismoving path is derived similarly as described above. When the wheel ofthe host vehicle 9 is turned at the position 9 c to move further towardthe available parking region P, the trace line derivation part 20 cderives a trace line based on the information of the moving path fromthe position 9 b to the position 9 c. In other words, the tracederivation part 20 c generates a trace line before turning the wheel byconnecting the moving path before turning the wheel.

The center line derivation part 20 a recognizes the available parkingregion P and derives a center line of the available parking region P. Inother words, the center line derivation part 20 a generates a lineextending in the parking direction over the approximate center part ofthe available parking region P. The prediction line derivation part 20 bderives a prediction line predicting a path of the host vehicle 9 inmoving from the position 9 c. In other words, the prediction linederivation part 20 b generates a line extending along the predicted pathfrom the center part of the host vehicle 9. The prediction of thismoving path is made similarly as described above.

The information superimposing part 23 generates a support image like thesupport image illustrated in FIG. 6 by superimposing and displaying thecenter line Lc, the prediction line Le, and the trace line Lt on aphotographic image photographed by the rear camera 5B. The displaycontroller 20 d makes the image output part 24 output the generatedsupport image to the display apparatus 3. The display apparatus 3displays this support image.

Even in a case where the wheel of the host vehicle has turned in parkingin a parking lot, etc., a trace line of the host vehicle before turningthe wheel and a prediction line after the turning the wheel aredisplayed. Hence, a difference of the traveling direction of the hostvehicle before and after turning the wheel can be specified. Bydisplaying the center line of the available parking region, it ispossible to support the host vehicle to park in the approximate centerof the available parking region.

<1-3. System Processing>

Next explained is a process performed by an image display system 10.FIGS. 7, 8, and 9 illustrate flowcharts showing a flow of the processperformed by the image display system 10.

The process performed by the image display system 10 starts by bootingup the image display system 10. When the process starts, the imagedisplay system 10 determines whether a switch is on or not (a step S10).The determination whether the switch is on or not is necessary in a casewhere a switch for displaying a support image by a user's manualinstruction is equipped. The determination whether the switch is on ornot is made based on operation information of the operation button 4 andthe like.

When the switch is on (Yes for the step S10), proceed to a displayprocess of a support image. On the other hand, when the switch is not on(No for the step S10), it needs to be determined whether a shift leveris in position reverse (R) or not (a step S11). The determinationwhether the shift lever is in position R or not is made based on asignal showing the shift position received from a shift sensor 92.

In a case where the shift lever is in position R (Yes for the step S11),there is a possibility of moving a vehicle backward to park. Therefore,proceed to the process of displaying a support image. In other words,when the shift lever is in R, proceed to the process of displaying asupport image automatically. On the other hand, in a case where theshift lever is not in R (No for the step S11), the process ends. It isbecause a display instruction of the support image is given neithermanually nor automatically when the switch is not on and the shift leveris not in R.

When the process proceeds to the process of displaying a support image,a center line derivation part 20 a performs a process of deriving acenter line (a step S12). The center line derivation part 20 a derivesan available parking region from a photographic image photographed by arear camera 5B. In other words, the center line derivation part 20 arecognizes a white line based on data of the photographic image andderives the available parking region by recognizing available spacethrough object detection such as a cubic object. However, the centerline derivation part 20 a may derive the available parking region usingother methods. When the center line derivation part 20 a derives theavailable parking region, a center line extending in a parking directionover an approximate center part of the available parking region isderived.

A prediction line derivation part 20 b performs a process of deriving aprediction line (a step S13). The prediction line derivation part 20 bderives a rotation direction and rotation angle of the host vehicle 9based on information acquired from a steering sensor 94 of the hostvehicle 9. The prediction line derivation part 20 b derives a predictionline predicting a moving path of the host vehicle 9 based on therotation direction and rotation angle of the host vehicle 9.

An information superimposing part 23 generates a support image (a stepS14). In other words, the information superimposing part 23 generatesthe support image by superimposing a center line and a prediction lineon a corresponding position of a photographic image photographed by therear camera 5B. The position of the center line on the photographicimage shows the approximate center of the available parking region. Theposition of the prediction line on the photographic image is thepredicted position on which the center of the host vehicle 9 will move.

When the information superimposing part 23 generates a support image, adisplay apparatus 3 displays the support image (a step S15). In otherwords, a display controller 20 d makes an image output part 24 outputthe support image generated by the information superimposing part 23 tothe display apparatus 3. The display apparatus 3 displays the supportimage input by the image output part 24. In this case, for example, thesupport image illustrated in FIG. 4 is displayed on the displayapparatus 3.

When the host vehicle 9 moves backward to the available parking region(a movement from a position 9 a to a position 9 b in FIG. 3), the imagedisplay system 10 stores the moving trace (a step S16). In other words,a controller 20 stores positional information acquired from a GPSreceiver 91 or a path on which the host vehicle 9 has moved based oninformation of a photographic image acquired by an image acquisitionpart 21 in a memory part 25.

The controller 20 determines whether the shift lever is in positionparking (P) or not (a step S17). It is determined whether the shiftlever is in position P or not based on a signal showing the shiftposition received from the shift sensor 92.

In a case where the shift lever is in position P (Yes for the step S17),the process ends because the shift lever in position P shows that thehost vehicle 9 stops. On the other hand, in a case where the shift leveris not in position P (No for the step S17), the controller 20 determineswhether the shift lever is in position drive (D) or not (a step S18).The determination whether the shift lever is in position D or not ismade based on a signal showing the shift position received from theshift sensor 92.

In a case where the shift lever is not in position D (No for the stepS18), the process is performed repeatedly from a process of storing amoving trace (the step S16) because the host vehicle is moving backward.On the other hand, in a case where the shift lever is in position D (Yesfor the step S18), it is determined that the wheel of the host vehicle 9has turned because the shift lever in position D shows that the shiftlever is changed from R to D. When the shift lever is changed between Rand D, it is determined that the wheel of the host vehicle 9 has turned.The information superimposing part 23 or the controller 20 may determinewhether the wheel of the host vehicle 9 has turned or not.

The image display system 10 performs a process of generating a supportimage. The process of generating the support image is performed by thederivation of a prediction line derived by the prediction linederivation part 20 b (a step S19). In other words, the prediction linederivation part 20 b derives the rotation direction and rotation angleof the host vehicle 9 according to information acquired from thesteering sensor 94 of the host vehicle 9. The prediction line derivationpart 20 b derives a prediction line predicting a moving path of the hostvehicle 9 based on the rotation direction and rotation angle of the hostvehicle 9.

The trace line derivation part 20 c performs a process of deriving atrace line (a step S20). In other words, the trace line derivation part20 c reads information of the moving path of the host vehicle 9 beforeturning the wheel from the memory part 25. The trace line derivationpart 20 c derives a trace line based on those read information.

The information superimposing part 23 generates a support image (a stepS21). In other words, the information superimposing part 23 generatesthe support image by superimposing a prediction line and a trace line ona corresponding position of a photographic image photographed by thefront camera 5F. A position of the prediction line on the photographicimage is a predicted position on which the center of the host vehicle 9will move. A position of the trace line on the photographic image is aposition on which the host vehicle 9 has moved before turning the wheel.

When the information superimposing part 23 generates the support image,the display apparatus 3 displays the support image (a step S22). Inother words, the display controller 20 d makes the image output part 24output the support image generated by the information superimposing part23 to the display apparatus 3. The display apparatus 3 displays thesupport image input by the image output part 24. In this case, forexample, the support image illustrated in FIG. 5 is displayed on thedisplay apparatus 3.

When the host vehicle 9 moves forward (a movement from the position 9 bto the position 9 c in FIG. 3), the image display system 10 stores themoving path (a step S23). In other words, the controller 20 stores thepath on which the host vehicle 9 has moved based on positionalinformation acquired from the GPS receiver 91 and information of thephotographic image acquired by the image acquisition part 21.

The controller 20 determines whether the shift lever is in position P ornot (a step S24). The determination whether the shift lever is inposition P or not is also made similarly as described above. In a casewhere the shift lever is in position P (Yes for the step S24), theprocess ends because the shift lever in position P shows that the hostvehicle 9 stops. On the other hand, in a case where the shift lever isnot in position P (No for the step S24), the controller 20 determineswhether the shift lever is in position R or not (a step S25). Thisdetermination whether the shift lever is in position R or not is madesimilarly as described above.

In a case where the shift lever is not in position R (No for the stepS25), the host vehicle 9 is moving forward. Therefore, the process isperformed repeatedly from a process of storing the moving trace (thestep S23). On the other hand, in a case where the shift lever is inposition R (Yes for the step S25), it is determined that the wheel ofthe host vehicle 9 has turned because the shift lever in position Rshows that the shift lever is changed from D to R. The image displaysystem 10 performs is the process of generating the support image. Theinformation superimposing part 23 or the controller 20 may determinewhether the wheel of the host vehicle 9 has turned or not.

The center line derivation part 20 a performs a process of deriving acenter line (a step S26). In other words, the center line derivationpart 20 a derives the available parking region based on a photographicimage photographed by the rear camera 5B. For derivation of theavailable parking region, the same method as described above is used.The center line derivation part 20 a derives the available parkingregion and a center line extending in the parking direction over theapproximate center part of the available parking region.

The prediction line derivation part 20 b performs a process of derivinga prediction line (a step S27). For this derivation of the predictionline, the same method as described above is used.

The trace line derivation part 20 c performs a process of deriving atrace line (a step S28). The trace line derivation part 20 c readsinformation of the moving path of the host vehicle 9 before turning thewheel from the memory part 25. The trace line derivation part 20 cderives a trace line based on those read information.

The information superimposing part 23 generates a support image (a stepS29). In other words, the information superimposing part 23 generatesthe support image by superimposing a center line, a prediction line, anda trace line on a corresponding position of a photographic imagephotographed by the rear camera B. A position of the center line on thephotographic image is a position showing the approximate center of theavailable parking region. A position of the prediction line on thephotographic image is a predicted position on which a center of the hostvehicle 9 will move. A position of the trace line is a position on whichthe host vehicle 9 has moved before turning the wheel.

When the information superimposing part 23 generates a support image,the display apparatus 3 displays the support image (a step S30). Forthis display, the same method as described above is used. In this case,for example, the support image illustrated in FIG. 6 is displayed on thedisplay apparatus 3.

When the host vehicle 9 starts to move backward to an available parkingregion (a movement to the available parking region P from the position 9c in FIG. 3), the image display system 10 repeats the process from theprocess of storing a moving trace (the step S16).

Thus, the image display system 10 in this embodiment is a system whichdisplays a support image superimposing a trace line before turning thewheel and a prediction line after turning the wheel on a photographicimage when the wheel of the host vehicle 9 has turned to park in aparking lot. The image display system 10 also superimposes a center lineon the photographic image when the host vehicle 9 moves to the availableparking region. The image display system 10 performs these processesrepeatedly until the host vehicle 9 stops.

By displaying a trace line and a prediction line, the moving path beforeand after turning the wheel can be displayed. Therefore, the user canspecifically understand the difference of the traveling direction of thehost vehicle. The display of the prediction line and the center lineenables the user to park in the available parking region easily.

2. Second Embodiment

A second embodiment is described. The first embodiment explains theconfiguration in which a host vehicle 9 moves backward to park. However,the invention is not limited to the first embodiment. This invention canbe applied to a case where the host vehicle 9 moves forward to park.Hence, the second embodiment explains a configuration in which the hostvehicle 9 moves forward to park focusing mainly on a point differentfrom the first embodiment.

FIG. 10 illustrates a situation in which the host vehicle 9 is beingparked in an available parking region P. The second embodiment shows anexample in which the host vehicle 9 moves forward to park. In otherwords, this is the example in which the host vehicle 9 moves from aposition 9 d to a position 9 e and then a wheel of the host vehicle 9 isturned so as to move the host vehicle 9 to a position 9 f, and furtherthe wheel of the host vehicle 9 is turned to park.

In this example, if the wheel of the host vehicle 9 is turned at theposition 9 e, from the position 9 d to the position 9 e is a movementbefore turning the wheel. From the position 9 e to the position 9 f is amovement after turning the wheel. In a case where the wheel of the hostvehicle 9 is turned at the position 9 f, from the position 9 e to theposition 9 f is a movement before turning the wheel. From the position 9f to a next position is a movement after turning the wheel.

<2-1. System Configuration and Process>

A configuration of an image display system in the second embodiment issimilar to a configuration of an image display system 10 in the firstembodiment explained in FIG. 1. Thus, detailed explanation on eachconfiguration is omitted. A process performed by the image displaysystem 10 in the second embodiment is explained below.

FIG. 11 illustrates a flowchart showing a flow of the process performedby the image display system 10 in the second embodiment. The processillustrated in FIG. 11 is a process corresponding to a forward movementand a backward movement which is mainly changed from a processcorresponding to a backward movement and a forward movement in the firstembodiment.

The process performed by the image display system 10 starts by bootingup the image display system 10. When the process starts, the imagedisplay system 10 determines whether a switch is on or not (a step S31).This embodiment has a configuration in which the host vehicle 9 movesforward to park. However, in order to distinguish a forward movement inparking and a normal driving, the process does not automatically proceedto a display process of a support image in a case where a shift lever isin position D, but the process proceeds to the display process of thesupport image in a case where an instruction is given by a usermanually.

Therefore, in a case where the switch is not on (No for the step S31),the process ends. On the other hand, in a case where the switch is on(Yes for the step S31), the process proceeds to a process of generatingand displaying the support image (a step S32). This process is same asthe steps S12, S13, S14 and S15 explained in the first embodiment.However, in this embodiment, the support image generated bysuperimposing a center line and a prediction line on a photographicimage photographed by a front camera 5F is displayed.

When the host vehicle 9 moves forward to an available parking region (amovement from a position 9 d to a position 9 e in FIG. 10), the imagedisplay system 10 stores the moving trace (a step S33). This process issame as the step S16 mentioned above.

A controller 20 determines whether the shift lever is in position P ornot (a step S34). In a case where the shift lever is in position P (Yesfor the step S34), the process ends because the shift lever in positionP shows that the host vehicle 9 stops. On the other hand, in a casewhere the shift lever is not in position P (No for the step S34), thecontroller 20 determines whether the shift lever is in position R or not(a step S35). A determination of a shift position explained in thisembodiment is made similarly as described above based on a signalshowing the shift position received from a shift sensor 92.

In a case where the shift lever is not in position R (No for the stepS35), the process is performed repeatedly from a process of storing amoving trace (the step S33) because the host vehicle 9 is movingforward. On the other hand, in a case where the shift lever is inposition R (Yes for the step S35), it is determined that a wheel of thehost vehicle 9 has turned because the shift lever in position R showsthat the shift lever is changed from D to R. Therefore, the imagedisplay system 10 proceeds to a process of generating and displaying asupport image (a step S36). This process is same as the steps S19, S20,S21 and S22 mentioned above. However, in this embodiment, a supportimage generated by superimposing a prediction line and a trace line on aphotographic image photographed by the rear camera 5B is displayed.Besides, similarly as mentioned above, the information superimposingpart 23 or the controller 20 may determine whether the wheel of the hostvehicle 9 has turned or not.

When the host vehicle 9 moves backward (a movement from the position 9 eto the position 9 f in FIG. 10), the image display system 10 stores themoving path (a step S37). This process is same as the step S23 mentionedabove.

The controller 20 determines whether the shift lever is in position P ornot (a step S38). In a case where the shift lever is in position P (Yesfor the step S38), the process ends because the shift lever in positionP shows that the host vehicle 9 stops. On the other hand, in a casewhere the shift lever is not in position P (No for the step S38), thecontroller 20 determines whether the shift lever is in position D or not(a step S39). In a case where the shift lever is not in position D (Nofor the step S39), the process is repeated from the process of storing amoving trace (the step S37) because the host vehicle 9 is movingbackward.

On the other hand, in a case where the shift lever is in position D (Yesfor the step S39), it is determined that the wheel of the host vehicle 9has turned because the shift lever in position D shows that the shiftlever is changed from R to D. Therefore, the image display system 10proceeds a process of generating and displaying a support image (a stepS40). This process is same as the steps S26, S27, S28, S29 and S30mentioned above. However, in this embodiment, the support imagegenerated by superimposing a center line, a prediction line, and a traceline on a photographic image photographed by the front camera 5F isdisplayed.

When the host vehicle 9 moves forward to the available parking region P(a movement to the available parking region P from the position 9 f inFIG. 10), the image display system 10 repeats the process from a processof storing a moving trace (the step S33).

The image display system 10 in this embodiment displays a support imagesuperimposing a trace line before turning the wheel and a predictionline after turning the wheel on a photographic image in parking the hostvehicle 9 in a parking lot with turning the wheel. Displaying the traceline and the prediction line makes it possible to display a moving pathbefore and after turning the wheel. Therefore, a user can specificallyunderstand a difference of the traveling direction of the host vehicle9. The image display system 10 displays a center line as well when thehost vehicle 9 moves toward the available parking region. The display ofthe prediction line and the center line enables the user to park in theavailable parking region easily.

3. Modifications

The embodiments of the invention have been described above. However, theinvention is not limited to the foregoing embodiments. This inventioncan be modified variously. Hereinafter, modification examples aredescribed below. Any of forms in the foregoing embodiments and in themodifications below may be combined with another appropriately.

In the first embodiment, an example is explained in which a supportimage is displayed when a host vehicle moves backward to park. In thesecond embodiment, an example is explained in which a support image isdisplayed when the host vehicle moves forward to park. A configurationmay be possible in which these examples are combined.

A concrete example of combination is as follows. An image display systemis configured to be able to perform any processes in the first andsecond embodiments. In a case where a switch is on, the image displaysystem determines a shift position and performs a process explained inthe first embodiment if a shift lever is in position R. In a case wherethe shift lever is in position D, the image display system performs aprocess explained in the second embodiment. Even in a case where theswitch is not on, a process of displaying a support image may beperformed automatically when the shift lever is in R.

In the foregoing embodiments, support information like a predictionline, etc. is superimposed on a photographic image photographed by arear camera 5B or a photographic image photographed by a front camera5F. However, the support information may be superimposed on a compositeimage generated by an image composite part 22.

For example, in a case where a virtual viewpoint whose position of aviewpoint is directly above the host vehicle 9 and whose gaze directionis directly below is set, the image composite part 22 generates acomposite image looking down at the host vehicle 9 and a region near thehost vehicle 9. An information superimposing part 23 generates a supportimage by superimposing support information like a prediction line, etc.on the composite image.

An explanation is given as follows with reference to Figs. In a casewhere the host vehicle 9 is in a position 9 a in FIG. 3, a support imageillustrated in FIG. 12 is generated by generating a support image by useof a composite image with a virtual viewpoint whose position of aviewpoint is directly above a rear portion of the host vehicle 9 andwhose gaze direction is directly below. In other words, the supportimage is an image superimposing a prediction line Le which predicts amoving path of the host vehicle 9 and a center line Lc which shows anapproximate center part of an available parking region P on thecomposite image viewing the rear portion of the host vehicle from above.

Likewise, in a case where the host vehicle 9 is in a position 9 b inFIG. 3, a support image illustrated in FIG. 13 is generated bygenerating a support image by use of a composite image with a virtualviewpoint whose position of a viewpoint is above a front portion of thehost vehicle 9 and whose gaze direction is directly below. In otherwords, the support image is an image superimposing a trace line Lt whichshows a moving path before turning the wheel and a prediction line Lewhich predicts a moving path after turning the wheel on the compositeimage viewing the front portion of the host vehicle 9 from above.

Likewise, in a case where the host vehicle 9 is in a position 9 c inFIG. 3, for example, a support image illustrated in FIG. 14 is generatedby generating a support image is by use of a composite image with avirtual viewpoint whose position of a viewpoint is above the rearportion of the host vehicle 9 and whose gaze direction is directlybelow. In other words, the support image is an image superimposing atrace line Lt which shows a moving path before turning the wheel, aprediction line Le which predicts a moving path after turning the wheel,and a center line Lc which shows a center part of the available parkingregion P on the composite image viewing the rear portion of the hostvehicle 9 from above.

As described above, by using different kinds of image data (such as aphotographic image and a composite image) for generation of a supportimage and generating plural kinds of composite images whose position ofa virtual viewpoint is different from each other, the support image canbe generated based on the images viewing the region near the hostvehicle 9 from various positions.

A support image generated by use of the photographic images illustratedin FIGS. 4, 5, and 6 and a support image generated by use of thecomposite images illustrated in FIGS. 12, 13, and 14 may be switched anddisplayed. In other words, the display is switched by a user selecting adesired display form when a support image is displayed. For example,when the support image illustrated in FIG. 4 is displayed, by operatingan operation button 4 or a touch panel, etc., it comes to be possible toswitch the display of the support image illustrated in FIG. 4 to thesupport image illustrated in FIG. 12. This switch may be performed whenthe process of displaying a support image starts or while the supportimage is displayed, the switch may be performed in arbitrary timing. Thedisplay form may be automatically switched depending on a drivingsituation, etc. of the host vehicle 9.

In the foregoing embodiments, a center line, a prediction line, and atrace line are used as support information. In addition to thisinformation, a guiding line showing a width of the host vehicle 9, etc.may be superimposed and displayed. When the guiding line is superimposedon the support images illustrated in FIGS. 4, 5, and 6, the supportimages illustrated in FIGS. 15, 16, and 17 are generated.

FIG. 15 illustrates an example of a support image superimposing aguiding line Lg on the support image illustrated in FIG. 4. FIGS. 16 and17 illustrate examples of support images superimposing a guiding line Lgon respective support images illustrated in FIGS. 5 and 6. A user mayswitch whether to superimpose this guiding line or not. The guiding linecan be displayed only if the user wants to display. The guiding line maybe superimposed and displayed on a support image by use of the compositeimages illustrated in FIGS. 12, 13, and 14.

As explained in the foregoing embodiments, this invention stores themoving trace of the host vehicle. By reading and displaying the storedmoving trace, it is possible to look back at a driving path of the hostvehicle later. For example, if the moving trace is displayed when thehost vehicle runs off while driving on a S-shaped bend or a crank road,a cause of the running off can be grasped.

In the foregoing embodiments, it is explained that various functions areimplemented as software by a performance of arithmetic processing of aCPU based on programs. However, a part of the functions may beimplemented by an electrical hardware circuit. A part of the functionsimplemented by the hardware circuit may be implemented as software.

While the invention has been shown and described in detail, theforegoing description is in all aspects illustrative and notrestrictive. It is therefore understood that numerous othermodifications and variations can be devised without departing from thescope of the invention.

What is claimed is:
 1. An image processing apparatus that processes animage, the image processing apparatus comprising: an image processorconfigured to: acquire a photographic image from a camera photographinga region near a host vehicle; derive a trace line showing a moving traceof a path on which the host vehicle has moved previously; derive aprediction line showing a predicted moving path of the host vehicle;generate a support image by superimposing the trace line and theprediction line on the photographic image when a wheel of the hostvehicle is turned; and output the support image to a display apparatusthat displays the support image.
 2. The image processing apparatusaccording to claim 1, wherein: the image processor is configured toacquire information as to a shift position of the host vehicle, and theimage processor determines whether the wheel of the host vehicle hasturned when the shift position changes between a position showing aforward movement of the host vehicle and a position showing a backwardmovement of the host vehicle.
 3. The image processing apparatusaccording to claim 1, wherein: the image processor is configured torecognize an available parking region where the host vehicle can beparked, and derive a center line showing a center part of the availableparking region, and the image processor generates the support image bysuperimposing the trace line, the prediction line, and the center lineon the photographic image.
 4. The image processing apparatus accordingto claim 1, wherein the photographic image is an image photographed by acamera photographing a rear portion of the host vehicle or a cameraphotographing a front portion of the host vehicle.
 5. The imageprocessing apparatus according to claim 1, wherein: the image processoris configured to generate a composite image showing a region near thehost vehicle viewed from a virtual viewpoint based on the photographicimage, and the image processor generates the support image bysuperimposing the trace line and the prediction line on the compositeimage when the wheel of the host vehicle is turned.
 6. The imageprocessing apparatus according to claim 5, wherein: the image processoris configured to recognize an available parking region where the hostvehicle can be parked, and derive a center line showing a center part ofthe available parking region, and the image processor generates thesupport image by superimposing the trace line, the prediction line, andthe center line on the composite image.
 7. The image processingapparatus according to claim 5, wherein the image processor outputs thesupport image generated by use of the photographic image and the supportimage generated by use of the composite image to the display apparatusalternately.
 8. The image processing apparatus according to claim 5,wherein the image processor generates the support image by furthersuperimposing a guiding line showing a width of the host vehicle on thephotographic image or on the composite image.
 9. An image display systemcomprising: the image processing apparatus according to claim 1; and adisplay apparatus that displays an image.
 10. An image processing methodthat processes an image, the method comprising the steps of: (a)acquiring, by an image processor, a photographic image from a cameraphotographing a region near a host vehicle; (b) deriving, by the imageprocessor, a trace line showing a moving trace of a path on which thehost vehicle has moved previously; (c) deriving, by the image processor,a prediction line showing a predicted moving path of the host vehicle;(d) generating, by the image processor, a support image by superimposingthe trace line and the prediction line on the photographic image when awheel of the host vehicle is turned; and (e) outputting, by the imageprocessor, the support image to a display apparatus that displays thesupport image.